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Abstract — The  vehicle  accident  prediction  methods  are  designed  to  improve 
the  vehicular  safety  and  reduce  the  rescue  response  time  in  the  case  of 
an  accident.  The  existing  accident  prediction  methods,  however,  do  not 
involve  Human-in-the-Loop,  i.e.,  do  not  consider  the  emotional  state  of  a 
driver  to  predict  the  likelihood  of  an  accident.  We  propose  a  Probabilistic 
Convolutional  Neural  Network  (CNN)-Fuzzy  Logic  framework  that  involves 
Human-in-the-Loop  and  takes  into  account  the  multiple  input  streams  of 
sensor  generated  data,  i.e.,  human  emotions  and  traffic  data.  The  features 
extracted  from  the  CNN  model  are  fed  to  our  designed  probabilistic  graph- 
based  inference  model  to  determine  the  accident  probability.  The  probability 
is  then  mapped  with  accident  severity  through  fuzzy  membership  functions 
for  accident  prediction.  The  experiment  results  show  the  promising  perfor¬ 
mance  of  our  proposed  framework,  i.e.,  93.1%  accuracy  of  face  expressions,  76.2%  accuracy  of  heartbeat,  and  76.9% 
accuracy  of  traffic  inputs  and  predicts  the  accident  likelihood  with  90%  accuracy.  The  comparison,  with  related  works, 
shows  that  the  proposed  framework  can  predict  accidents  with  higher  probabilities. 

Index  Terms — Accident  Prediction;  Convolution  Neural  Network;  Deep  Learning;  Fuzzy  Logic;  Human-in-the-Loop. 
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I.  Introduction 

RECENT  advancements  in  sensing  and  actuating  technol¬ 
ogy,  along  with  the  developments  of  innovative  intelli¬ 
gent  solutions,  have  realized  the  realm  of  Intelligent  Trans¬ 
portation  System  (ITS)  [1]— [3].  ITS  has  a  number  of  input 
streams  of  data  coming  through  various  sources  such  as  traffic 
control  systems,  vehicular  sensors,  and  driver-generated  data 
to  offer  next-generation  vehicular  services  such  as  maintaining 
traffic  efficiency,  offering  safety  and  comfort  to  drivers  and 
passengers,  fleet  management,  and  many  others  [4].  Vehicular 
safety  against  accidents  is  an  imperative  aspect  that  must  be 
considered  while  designing  any  ITS  service.  According  to 
World  Health  Organization,  a  large  proportion  of  deaths  and 
injuries  all  over  the  world  are  due  to  road  accidents  [5].  In 
many  cases,  the  deaths  are  due  to  lack  of  coordination  with  the 
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vehicles  or  lack  of  timely  alerts  generated  by  the  coordinated 
systems  for  vehicles.  Thus,  there  is  a  need  for  a  robust  accident 
alerting  framework  for  improving  the  road  safety  by  taking 
into  account  different  streams  of  data. 

The  underlying  sensors,  however,  may  generate  imprecise 
values  in  cyber-physical  systems  in  general,  and  ITS,  in 
particular,  along  with  inherited  randomness  involved  in  the 
transportation  data  [6]— [8] .  The  researchers  from  academia  and 
industry  has  explored  deep  learning,  graph  theory,  fuzzy  logic, 
and  other  soft  computing  domains  to  design  different  safety 
solutions  in  vehicular  networks  [9]— [12]. 

The  existing  deep  learning-based  works  have  primarily 
focused  on  the  accident  detection  and  recovery  by  mainly 
considering  traffic  flow,  travel  time  estimation,  and  predicting 
modes  of  transport  [9],  [10].  Probabilistic  graph  models  are 
used  for  analyzing  the  hidden  relationships  between  the  vari¬ 
ous  elements  in  unstructured  data  [10].  The  graph  based  repre¬ 
sentation  is  used  for  encoding  and  factorizing  representation  of 
independence  in  distribution  space.  Similarly,  the  probabilistic 
and  fuzzy  logic-based  models  are  studied  to  solve  the  vehicle¬ 
centric  safety  problem  [11],  [12].  Most  of  these  existing 
solutions,  however,  mainly  consider  external  information  (i.e., 
traffic  status,  vehicle  speed,  etc.,).  The  internal  emotional  state 
of  the  driver  is  not  considered  in  these  studies.  The  auto¬ 
matic  prediction  of  emotion  promises  to  revolutionise  human- 
computer  interaction;  thus,  offering  more  reliable  services  in 
realms  such  as  ITS.  Recent  trends  involve  fusion  of  multiple 
data  modalities  -  audio,  visual,  and  physiological  -  to  predict 
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Fig.  1 :  Accident  prediction  framework 


emotional  state  [13].  The  accurate  prediction  of  emotions  of 
drivers  may  play  significant  role  in  avoiding  traffic  accidents. 

To  bridge  the  above-cited  research  gaps,  this  study  aimed  at 
designing  and  analyzing  a  probabilistic  convolutional  Neural 
Network  (CNN)-Fyzz  Logic  framework  to  effectively  predict 
accidents  using  both  external  and  internal  parameters.  The 
fuzzy  logic  embodies  the  degree  to  which  outcome  of  the 
graph-based  inference  model  is  associated  with  driver  alerts. 
As  a  first  step,  we  used  Kaggle’s  facial  expression,  heartbeat, 
and  traffic  data  set.  Then  we  use  CNN  model  to  classify 
emotion,  abnormal  heartbeat,  and  traffic  situation.  This  is 
followed  by  designing  a  probabilistic  graph-based  inference 
model  to  predict  the  accident  probability  by  extracting  com¬ 
plex  relationships  between  drivers  internal  information  and 
external  traffic  information  for  inference.  Finally,  the  accident 
probability  is  mapped  to  accident  severity  to  generate  alerts 
to  the  driver.  The  experiments  show  the  promising  results  of 
our  proposed  framework.  The  key  novelties  of  our  proposed 
framework  are  summarized  below: 

1)  We  designed  a  deep  learning-based  model  that  considers 
both  internal  state  of  the  driver  and  external  states  of 
vehicle  and  traffic  for  accident  prediction.  The  internal 
state  is  determined  through  facial  expressions  and  heart¬ 
beat  status  of  drivers.  The  external  states  are  determined 
through  vehicular  and  traffic  data  parameters. 

2)  The  output  of  above  model  is  fed  to  our  proposed 
probabilistic  graph-based  inference  model  to  determine 
the  accident  probabilities. 

3)  Finally,  we  defined  fuzzy  logic-based  rule  base  mapping 
to  map  accident  probability  to  severity  of  the  accidents 
to  generate  alerts  to  drivers. 

The  rest  of  the  paper  is  structured  as  follows:  Section  II 
elucidates  the  related  work.  The  proposed  framework  is  de¬ 
scribed  in  Section  III.  Performance  of  the  presented  framework 
is  evaluated  in  Section  IV.  Section  V  concludes  the  findings 
of  this  work. 


II.  Related  Work 

Li  et  al  [14]  proposed  an  accident  prediction  model  by 
exploiting  multi-dimensional  time  series  information.  The  grey 
correlation  was  studied  to  assess  the  influence  of  time  series 
of  each  factor- variable  on  each  accident  causing  variable.  The 
closeness  centrality  measure  was  computed  to  measure  the 
degree  of  correlation  amongst  the  factor  and  accident  variables. 
Zheng  et  al  [15]  proposed  a  hybrid  deep  learning-based 
model,  Conv-LSTM,  to  predict  traffic  flows,  based  on  CNN 
and  the  long  short-term  memory  network.  The  model  extracts 
the  intrinsic  characteristics  of  the  data  of  traffic  flows  and 
mines  the  spatio-temporal  characteristics  of  traffic  flow.  Wu 
et  al  [16]  proposed  a  similar  model  by  using  CNN  to  extract 
the  spatial  characteristics  of  traffic  flow  and  designed  LSTM 
for  extracting  the  dynamic  characteristics  of  traffic  flows. 

A  Markov  model  was  presented  to  predict  accidents  based 
on  spatial  and  temporal  distribution  characteristic  of  acci¬ 
dents  [17].  The  correlations  between  road  type,  traffic  flow, 
and  traffic  accident  were  derived.  The  network  latency  mea¬ 
surement  was  estimated  using  multimodel  deep  learning  for 
the  selection  of  a  server  [18].  The  authors  studied  the  impact 
of  driver  stress  and  fatigue  in  car  accidents  [19].  This  indicates 
the  lack  of  multimodel  techniques  in  this  research  domain. 

The  existing  models,  however,  do  not  consider  the  emo¬ 
tional  state  of  drives.  Most  of  the  previous  works  considered 
passive  vehicle  dynamics  such  as  air  bag,  automatic  breaking 
system,  and  inter-vehicle  distance  that  have  improved  safety 
to  a  certain  degree.  Another  research  dimension  has  explored 
the  possibility  of  designing  algorithms  considering  real  time 
parameters  such  as  traffic  status,  vehicle  speed,  etc.,  [21]. 
However,  the  primary  limitation  of  the  current  deep  learning 
models  is  non-consideration  of  emotional  state  of  drivers. 
Research  shows  human  error  plays  a  significant  role  in  car 
accidents.  Particularly,  by  novice  drivers  in  the  complex  road 
environments  [20].  Thus,  designing  a  method  that  takes  into 
account  the  emotional  state  of  drivers  and  road  traffic  is  desired 
to  timely  generate  alerts  to  prevent  car  crashes. 
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TABLE  I:  Notations 


Symbol 

Description 

Symbol 

Description 

es 

smile 

th 

high  traffic 

en 

neutral 

P(A)  or  PA 

accident  probability 

ea 

angry 

FI 

Facial  Images 

hn 

normal  heartbeat 

HB 

Heartbeat 

ha 

abnormal  heartbeat 

TF 

Traffic  Flow 

tl 

low  traffic 

D 

input  data 

tm 

medium  traffic 

O 

observed  evidences 

III.  Proposed  Framework 

The  accident  prediction  framework  has  a  number  of  phases, 
namely,  data  retrieval,  features  extraction,  probabilistic  graph- 
based  inference  modeling,  mapping  rules,  and  generating 
results,  as  shown  in  Fig.  1.  The  key  notations  are  given  in 
Table  I. 

A.  Accident  Prediction  Method 

1)  Features  Extraction:  The  CNN  model  is  formulated  for 
features  extraction  [30].  The  input  data  to  the  CNN  model 
is  emotion  data  of  drivers  (also  called  drivers  internal  data 
in  this  case)  and  external  traffic  information.  These  features 
are  given  as  input  to  the  inference  model  to  derive  mean¬ 
ingful  conclusions.  There  are  two  layers  of  the  CNN  model: 
forward  and  backward  transmission.  In  forward  transmission, 
internal  and  external  information  are  passed  as  input  to  the 
convolution  layers  and  output  is  obtained.  In  general,  there 
are  n  input  matrices,  X{  represents  the  ith  input  matrix,  and  C{ 
represents  the  ith  sub-convolution  matrix.  Equation  (1)  shows 
the  convolution  block  in  CNN  [30]. 

n 

v™v{x)  =  f(WTx)  =  fC£wiXc  +  b)  (1) 

i=  1 

where  WT  is  the  matrix  in  convolution  layer,  x  and 
VwbV(x)  are  inPut  and  output  of  the  convolution  layer, 
respectively.  The  notation  b  is  the  deviation  and  /  is  the 
activation  function. 

The  residual  function  used  to  solve  the  saturation  problem 
in  the  activation  function  is  given  by  (2)  [30]. 

Vresi  =  X1-1  +  01-1)  (2) 

where  xl~x  and  y'rcsl  are  the  current  input  and  output, 
respectively.  The  error  between  output  and  actual  values  is 
computed. 

2)  Input  Data  Description:  The  input  data  is  derived  from 
driver’s  facial  expressions,  heartbeat,  and  traffic  status. 

%type  •  %  expression  ^heartbeat  i  %traf  fic— flow  (3) 

In  the  CNN  model,  small  changes  in  the  input  can  cause 
disruption  in  the  output.  To  address  this  issue,  the  activation 
function  is  used.  The  activation  function,  /,  has  the  saturation 
problem.  To  solve  the  saturation  problem,  the  residual  function 
is  used.  The  complete  CNN  model  is  defined  as  given  below 

xl  =  xi~1  +  5(xi-1,0i-1)  (4) 


where  i  represents  layer,  S  is  error  function,  and  0  is  the 
parameter  related  to  the  previous  layers.  The  model  gets  better 
using  the  learning  parameters  and  reduces  the  error. 

L{e)  =  \\-Yt-YX  (5) 

3)  Algorithm:  The  data  is  generated  by  the  cooperative 
vehicular  infrastructure.  The  internal  (emotional)  driver  and 
the  external  vehicular  network  data  is  derived.  Both  data 
streams  are  then  fed  to  the  Algorithm  1  CNN  model  to  train 
it  until  the  optimal  parameters  are  achieved.  Then  predictions 
from  the  test  data  are  derived.  The  CNN  model  gives  outputs, 
namely,  driver  expressions:  es  denotes  smile,  en  denotes 
neutral,  and  ea  denotes  angry.  The  driver  heartbeat:  hn  denotes 
normal  and  ha  denotes  abnormal.  Finally,  the  traffic  status:  ti 
denotes  low  traffic,  tm  denotes  medium  traffic,  and  th  denotes 
high  traffic.  These  are  given  as  input  to  Algorithm  2,  which 
returns  conditional  probabilities  of  all  input  features  from  these 
probabilities.  The  accident  probability  P(A)  is  then  inferred 
and  mapped  to  the  accident  severity  value.  The  inferred  P(A) 
is  then  mapped  to  accident  severity  with  the  membership 
functions  using  (6)  t  (8).  Finally,  the  alerts  are  generated 
to  alert  drive  of  any  danger.  The  algorithmic  description 
of  the  complete  accident  prediction  process  is  provided  in 
Algorithm  1. 


Algorithm  1  Accident  prediction  based  on  internal  and 
external  data _ 

Input:  Facial  Images  (FI),  Heartbeat  (HB),  Traffic  Flow  (TF) 
Output:  Alerts 

l:  For  all  Xi  E  i[l,  t\  time  series  Do 
2:  Construct  input  data  using  (3) 

3:  Store  D  <-  FI ,  HB ,  TF 

4:  End  For 
5:  Repeat 

6:  Select  randomly  Di  e  D 
7:  Execute  CNN  using  (1)  and  (2) 

8:  Minimize  error  6  using  (5) 

9:  Stop  at  stopping  criteria 

10:  P(0\Pa(0))=  Alg  2(es,  en,  ea,  /in,  /ia,  tm,  s) 

ll:  Return  P(A). 

12:  Mapping  Safe ,  Cautious ,  Danger  <—  P(A) 

13:  Return  Alerts 


B.  Accident  Inference  Model 

1)  Model  Description:  Fig.  2  shows  the  graphical  represen¬ 
tation  of  the  extracted  features,  where  nodes  in  the  graph 
represent  features  and  edges  represent  the  dependent  proba¬ 
bilities  between  features.  This  model  infers  the  possibility  of 
accidents  based  on  the  internal  and  external  features  in  the 
vehicular  environment.  The  internal  features  include  driver 
expressions:  es  denotes  smile,  en  denotes  neutral,  and  ea 
denotes  angry.  The  driver  heartbeat:  hn  denotes  normal  and 
ha  denotes  abnormal.  Finally,  the  traffic  status:  ti  denotes  low 
traffic,  tm  denotes  medium  traffic,  and  th  denotes  high  traffic. 
The  algorithmic  description  is  shown  in  Algorithm  2. 
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Fig.  2:  Graph-based  accident  inference  model 


2)  Accident  Inference:  We  infer  the  probability  of  the  ac¬ 
cident  (PA)  from  the  outcomes  of  the  CNN  model  by  using 
them  as  input  to  the  graph-based  prediction  model.  We  find 
PA  as  an  outcome  from  the  model  which  has  three  categories 
or  alerts:  safe,  cautious,  and  danger.  The  inference  model 
is  employed  to  find  values  of  hidden  nodes  (Dn,Da,PA) 
for  each  instance  of  the  CNN  outcomes.  After  executing 
the  algorithm,  we  choose  max(PA)  as  the  inferred  category. 
We  map  PA  into  severity  of  accidents,  and  based  on  the 
membership  functions,  the  driver  will  be  given  alerts  as  safe, 
cautious,  or  danger.  Table  II  shows  the  relationship  between 
PA,  range  of  variables,  and  corresponding  linguistic  variables. 


Algorithm  2  Graph-based  accident  inference  algorithm 

Input:  CNN  outputs,  Observed  evidences 
Output:  Conditional  probability  P(0|Pa(0))  of  each  node 
Randomly  initialize  P(0|Pa(0)) 

2:  For  each  node  O 

While  P(0\Pa(0))  does  not  converge 
4:  For  Each  evidence  e  =  (es,  en,  ea,  ftn,  ha,  ti,  th,  tm,  s)  e 
E 

For  Each  value  of  ft,  =  (Dn,Da,  AP )  e  H 
6:  P(ft,  e)  <  P(Dn |en,  £/,  es,  hn)P(Da |ea,  t/i,  £m,  na) 
xP(AP\Dn,Da,S ) 

8:  For  Each  value  of  ft  =  (Dn,  Da,  AP)  e  H 
P(ft|e)  <  -P(h,e)\\Eh  e  HP(h,e) 

10:  End  For 
End  For 
12:  End  For 
End  While 
14:  End  For 

For  Each  node  O 

16:  7  <  —  the  occurrence  of  (0,Pa(0)) 

P(0\Pa(0)  <  —7  ||  the  occurrence  of  Pa(0) 

18:  End  For 

Return  P(0\Pa(0)) 

3)  Membership  Function:  Accident  alert  severity  groups  are 
shown  as  fuzzy  numbers  and  associated  membership  functions: 
“Safe”,  “Cautious”,  and  “Danger.  In  this  setting,  the  severity 
group  “Safe”  has  a  probability  of  accidents  from  0  to  0.37 
with  lower  limit  a  =  0,  support  limit  b  =  0.23,  and  an  upper 
limit  c  =  0.37.  The  severity  group  “Cautious”  has  probabilities 
lower  limit  b  =  0.23,  a  lower  support  limit  c  =  0.37,  an 


TABLE  II:  Accident  probability  illustrated  as  fuzzy  numbers 


Probability  Range  Linguistic  Variables 

P(A)  <  0.37  Safe 

Probability  of  Accident  P(A)  P(A)  >  °-23  and  P(A)  <  0.67  Cautious 

P(A)  >  0.53  Danger 


Fig.  3:  Membership  function 


upper  support  limit  d  =  0.53,  and  an  upper  limit  e  =  0.67. 
The  severity  group  “Danger”  has  probabilities  lower  limit  d 
=  0.53,  support  limit  e  =  0.67,  and  an  upper  limit  f  =  1.  The 
membership  in  the  severity  group  “Safe”  descends  gradually 
from  accident  probabilities  0.23  to  0.37.  The  solid  lines  show 
the  fuzzy  membership  functions  for  each  group.  The  dashed 
lines  denote  how  “Safe”  values  can  be  made  into  non-fuzzy 
crisp  values. 

The  description  of  vagueness  in  the  accident  severity  groups, 
i.e.,  a  lower  limit,  an  upper  limit,  a  lower  support  limit,  and 
an  upper  support  limit,  can  be  best  modeled  with  trapezoidal 
function.  Based  on  the  probability  of  the  accident,  the  ac¬ 
cident  severity  group  “Safe”  is  defined  as  a  special  case  of 
the  Trapezoidal  function  called  R— function  with  parameters: 
lower  limit  a  =  0,  support  limit  b  =  0.23,  and  upper  limit  c 
=  0.37,  where  a  <  b  <  c.  The  “Cautious”  group  is  defined 
as  a  Trapezoidal  function  with  probabilities:  lower  limit  b  = 
0.23,  a  lower  support  limit  c  =  0.37,  an  upper  support  limit  d 
=  0.53,  and  an  upper  limit  e  =  0.67,  where  b  <  c  <  d  <  e. 
Finally,  “Danger”  is  defined  as  another  special  case  of  the 
Trapezoidal  function  called  F-function  with  parameters  lower 
limit  d  =  0.53,  support  limit  e  =  0.67,  and  an  upper  limit  f  = 
1,  where  d  <  e  <  f .  Fig.  3  shows  the  membership  functions 
of  “Safe”,  “Cautious”,  and  “Danger”. 

Let  us  consider  the  accident  probability  as  Universe  of 
Discourse,  representing  severity  of  accidents  as  three  groups: 
“Safe”,  “Cautious”,  and  “Danger”.  The  corresponding  mem¬ 
bership  functions  are  defined  below. 

For  safe,  if  x  >  c,  the  corresponding  function  value  is  zero. 
If  b  <  x  <  c,  then  the  corresponding  value  is  calculated 
using  equation  of  line  segment  Finally,  if  x  <  b  the 
corresponding  function  value  is  1.  The  function  for  safe  is  as 
follows. 

{0,  x  >  c 

fcf,  b<x<c  (6) 

1,  x  <  b 

For  cautious,  if  x  <  b,  the  corresponding  function  value  is 
zero.  If  b  <  x  <  c,  then  the  corresponding  value  is  calculated 
using  the  equation  of  the  line  segment  joining  the  two  points 
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b  and  c.  We  get  For  c  <  x  <  d,  the  corresponding 
function  value  is  1.  For  d  <  x  <  e,  the  function  is  again  the 
line  segment  between  points  d  and  e.  We  get  .  Finally,  for 
x  >  e,  the  function  value  is  zero.  The  function  for  Cautious 
is  as  follows. 


{0,  (x  <  b )  or  (x  >  e) 

r6’  -  (T 

fEi,  d<x<e 

For  “Danger”,  if  x  <  d,  the  corresponding  function  value 
is  zero.  If  the  d  <  x  <  e,  then  the  corresponding  value 
is  computed  using  equation  of  the  line  segment  joining  two 
points  d  and  e.  We  get  Finally,  if  x  >  e  the  corresponding 
function  value  is  1.  The  function  for  danger  is  as  follows. 


ft D anger  {X')  —  < 


x—d 
e—d  ’ 


X  <  d 

d  <  x  <  e 
x  >  e 


(8) 


Each  group  in  the  severity  of  accidents,  i.e.,  “Safe”,  “Cau¬ 
tious”,  and  “Danger”  have  a  crisp  membership  value  function. 
Once  the  max(PA)  has  been  inferred  from  the  probability 
inference  model,  this  probability  is  then  mapped  to  related 
severity  of  accidents  member.  If  PA  falls  in  the  region  “Safe”, 
then  driver  gets  a  safe  indication.  If  it  falls  in  “Cautious” 
region,  then  the  driver  gets  cautious  indication.  If  it  falls  in 
the  region  “Danger”,  then  the  driver  gets  Danger  indication 
so  that  the  driver  will  take  appropriate  measures  to  avoid  any 
accident. 

The  detailed  description  is  provided  below: 

1)  Initialization  of  the  parameters  for  the  CNN  model.  The 
connection  weights  and  deviation  values  are  initialized. 

2)  The  extraction  of  three  types  of  information.  We  use  three 
types  of  information  as  input:  face  expression  (typel), 
heartbeat  (type2),  and  traffic  status  (type3),  as  shown 
in  (3). 

3)  The  CNN  model  consists  of  a  series  of  consecutive 
convolution  and  residual  blocks. 

a)  The  convolution  block,  an  activation  function,  is  uti¬ 
lized  to  consider  the  non-linear  factor. 

b)  The  residual  block  solved  the  problem  of  gradient  dis¬ 
sipation  caused  by  the  inability  to  update  the  saturation 
network  weight. 


6(x,  0)  =  W2cj{W1x  +  bi)  +  b2  (9) 

where  W\ ,  W2  and  b\ ,  b2  are  the  weights  and  the  de¬ 
viations,  respectively.  The  notation  a  is  the  nonlinear 
activation  function. 

4)  CNN  is  trained  by  back  propagating  loss  and  Gradient 
decent  value  until  the  stop  criteria  is  attained. 

5)  The  reverse  fine-tuning  is  performed. 

a)  The  loss  function  is  calculated  using  (5). 

b)  The  stochastic  gradient  decent  value  gives  loss  at  each 
layer  between  input  and  output.  This  is  calculated  using 
the  following  equation  [31] 


TABLE  III:  Performance  with  different  datasets 


CNN  - 
Emotion 

Data 

CNN  - 
Heart-Beat 

Data 

CNN- 

Traffic  Status 

Data 

Train 

Test 

Train 

Test 

Train 

Test 

Accuracy 

94.4% 

93.1% 

76.6% 

76.2% 

94.2% 

76.9% 

Sensitivity 

79.0% 

81.1% 

83.0% 

87.2% 

79.1% 

83.8% 

Precision 

31.0% 

21.1% 

19.0% 

19.2% 

29.8% 

19.2% 

FI  score 

44.7% 

33.8% 

31.3% 

31.8% 

43.5% 

31.6% 

,i  .  =  dL_  =  dL  %-f1  dx-f 1 
ij  ~  96Z  ~  dyiP  dx\p  dOZ 


(10) 


where  0  includes  w  and  b ,  L  is  the  loss  function,  l  and 
l  + 1  correspond  to  the  current  and  next  layers,  Xij  and 
Uij  represent  input  and  output, 

c)  Then  update  the  weights  by  back  the  propagating  error 
[31] 


dl 

eij  =  e\j  ~  w(#y )  =  Q\j  -g-Q^r  T i) 

^3 

where  r]  is  the  learning  rate,  set  to  0.0002. 

6)  Train  the  CNN  model  until  the  early  stopping  strategy  is 
achieved. 

7)  Test  using  the  trained  optimal  CNN  model. 

8)  Graph-based  inference  is  derived  using  Algorithm  2.  The 
graph  is  constructed  to  find  the  hidden  nodes  Dni  Da,  and 
P(A)  from  the  outcomes  of  the  CNN  model.  The  con¬ 
ditional  probability  of  each  node  is  initialized  with  zeros 
and  the  EM  algorithm  is  applied  until  the  probability  of 
the  hidden  nodes  is  converged.  The  outcome  we  consider 
here  is  max(P(A)). 

9)  The  accident  probabilities  are  assigned  memberships  in 
three  groups  “Safe”,  “Cautious”,  and  “Danger”. 

10)  The  max(P(A))  inferred  from  graph-based  inference 
method  is  mapped  into  ’’Safe”,  ’’Cautious”  and  ’’Danger” 
groups: 

a)  If  x  =  p  and  if  p  =  z  and  if  z  is  in  Safe  then  the  Safe 
alert  is  given  to  the  driver. 

b)  If  x  =  p  and  if  (i  —  z  and  if  z  is  in  Cautious  then 
Caution  alert  is  given  to  the  driver. 

c)  If  x  =  p  and  if  fi  =  z  and  if  z  is  in  Danger  then 
Danger  alert  is  given  to  the  driver,  where  notation  x 
is  inferred  probability,  p  is  inferred  probability  value, 
(i  is  membership  of  inferred  probability,  and  2  is  the 
membership  value  of  inferred  probability. 

11)  Using  the  above  rule  set  the  membership  of  the  probabil¬ 
ity  is  mapped  to  the  severity  group,  the  groups  “Safe”, 
“Cautious”,  or  “Danger”,  whichever  is  found,  the  corre¬ 
sponding  alert  is  issued  to  the  driver. 

If  the  mapped  severity  group  is  “Safe”,  then  the  driver 
will  have  an  indication  to  continue  with  driving.  If  the 
mapped  severity  group  is  “Cautious”,  then  the  driver  will 
have  an  indication  to  slow  down  the  vehicle.  Finally,  if 
the  mapped  severity  group  is  “Danger”,  then  the  driver 
will  have  an  indication  to  stop  the  vehicle  to  avoid  an 
accident. 
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TABLE  IV:  Test  data  with  various  ranges  of  accident  prediction 


S.No 

Observed 

Predicted 

Absolute  error 

67 

40.23 

47.7 

7.47 

68 

75.025 

89 

13.975 

69 

8.16 

9.6 

1.44 

83 

47.25 

47.7 

0.45 

84 

13.32 

3.6 

9.72 

85 

10.29 

22.5 

12.21 

86 

18.57 

20.7 

2.13 

87 

80.55 

65.7 

14.85 

88 

30.06 

26.7 

3.36 

89 

25.05 

19.8 

5.25 

96 

10.29 

9.6 

0.69 

97 

1.05 

1.8 

0.75 

98 

19.5 

21.6 

2.1 

99 

4.86 

4.5 

0.36 

100 

7.08 

9.6 

2.52 

114 

36.81 

36.6 

0.21 

115 

61.44 

47.7 

13.74 

116 

30.93 

21.6 

9.33 

117 

3.84 

0 

3.84 

118 

14.46 

20.7 

6.24 

119 

17.31 

0 

17.31 

IV.  Performance  Evaluation 

A.  Experiment  Setup 

1)  Datasets:  We  have  used  three  datasets:  emotion,  heart¬ 
beat,  and  traffic  datasets  from  the  Kaggle  website  [22].  These 
three  datasets  are  given  as  the  input  to  the  CNN  (see  Fig.  1). 
The  emotion  dataset  contains  35,800  gray  scale  pictures, 
where  each  image  belongs  to  one  of  the  following  cate¬ 
gories:  “angry”,  “happy”,  and  “neutral”.  The  heartbeat  dataset 
is  taken  from  the  PTB  diagnostic  ECG  database  which  is 
pre-processed  by  Kaggle  [22].  It  contains  14,552  samples, 
which  belong  to  the  normal  and  abnormal  categories.  For 
traffic  information,  we  used  the  PeMS  dataset  prepocessed  by 
Kaggle  [22].  the  PeMS  dataset  is  the  widely  used  dataset  for 
traffic  prediction  [23],  where  data  belong  to  three  categories: 
“high”,  “medium”,  and  “low”. 

B.  Performance  Metrics 

To  evaluate  the  performance  of  the  formulated  CNN  models, 
we  have  used  the  standard  performance  measures,  namely, 
Accuracy,  Sensitivity,  Precision,  and  FI  score  [24]. 


TP  +  TN 

ccuracy  -  Tp  +  TjV  +  pN  +  pp 

(12) 

„  ...  TP 

Sensitivity  =  Tp  +  pN 

(13) 

Tp 

Precision  =  — — - — — 

TP  +  FP 

(14) 

sensitivity  x  precision 

- ensitimty  +  precision 

o 

(15) 

Fig.  4:  Training  phase:  the  predicted  and  observed  accident 
probabilities 


100  200  300  400  500  600  700  800  900  1000 

Number  of  Samples 


Fig.  5:  Testing  phase:  the  predicted  and  observed  accident 
probabilities 


C.  Results  and  Analysis 

The  proposed  framework  was  run  with  the  three  datasets:  (i) 
emotion  dataset,  (ii)  traffic  dataset,  and  (iii)  heartbeat  dataset. 
These  three  datasets  were  split  into  training  and  testing  sets. 
The  model  that  was  built  using  the  training  data  was  then  fed 
by  the  testing  data  to  evaluate  the  overall  performance  of  the 
model.  The  performance  of  the  CNN  model  was  evaluated 
based  on  the  above-cited  metrics. 

Table  III  describes  the  performance  of  the  CNN  model 
before  and  after  training.  The  results  show  the  model  tend 
to  perform  better  with  93.1%  accuracy  by  emotion  dataset 
followed  by  76.9%  accuracy  by  the  traffic  dataset,  and  76.2% 
accuracy  by  the  heartbeat  dataset.  The  probabilistic  graph- 
based  model  predicts  probabilities  and  these  probabilities  are 
compared  with  the  observed  probabilities  to  find  the  error 
caused  by  the  proposed  framework.  Table  IV  shows  the 
observed  and  predicted  probabilities,  and  associated  computed 
error.  Fig.  4  and  Fig.  5  provide  graphical  views  of  the 
differences  between  the  probabilities  for  training  and  test 
data  samples.  The  proposed  framework  predicted  accident 
probabilities  that  are  mapped  to  severity  of  accidents  which 
will  in  turn  generate  a  corresponding  alert  to  the  driver. 

We  have  compared  the  predictions  of  our  proposed  frame¬ 
work  with  two  models:  modell  [25]  and  model2  [26].  Fig.  6 
shows  the  comparative  prediction  probabilities.  The  graph 
shows  that  our  framework  tends  to  perform  better  as  compared 
to  the  other  models.  This  is  due  to  the  involvement  of  human 
emotions  in  the  whole  process.  The  deep  learning  models 
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are  more  dependent  on  the  datasets  and  these  datasets  can 
be  unbalanced  leading  to  lack  of  applicability  in  critical 
domains  (i.e.,  medical,  etc)  [32]-[34].  Adding  probabilistic 
considerations  to  the  deep  learning  models  improves  the  ap¬ 
plicability  of  effective  computing  in  environments  particularly 
when  considering  human  emotions  for  decision  making.  In  the 
previous  models,  the  accident  prediction  is  based  on  vehicle 
external  information,  i.e.,  traffic  status,  but  in  our  model 
we  have  included  the  vehicle  internal  status  by  considering 
driver’s  emotion  with  face  expressions  and  driver’s  heartbeat. 
If  this  internal  status  of  the  driver  is  abnormal,  then  the  chance 
of  accident  is  more.  Hence,  there  are  high  peaks  of  the  green 
line  in  Fig.  6.  We  have  used  a  probabilistic  decision  making 
method,  i.e.,  graph-based  inference  method  that  takes  both 
vehicle  internal  and  external  status,  and  identifies  probability 
of  accidents.  Since  our  model  is  based  on  both  internal  and 
external  information  with  probabilistic  decision  making,  the 
results  vary  with  other  models  (i.e.,  model  1  and  model2). 

In  a  previous  work  [27],  a  single  dataset  was  used  for 
traffic  analysis  and  modeling.  In  our  framework,  we  used  three 
datasets  which  reflect  better  results  in  determining  accident 
probability.  Some  other  previous  works  used  multiple  datasets, 
but  they  did  not  consider  the  internal  information  of  vehicles 
such  as  driver  emotions:  heartbeat  and  facial  expression  that 
could  affect  on  the  state  of  the  vehicle  and  can  cause  accident 
[28],  [29].  This  shows  that  the  driver  situation  has  more  prob¬ 
abilistic  affect  on  accidents.  Thus,  more  accurate  prediction  of 
accidents  and  generation  of  appropriate  alerts  can  reduce  the 
number  of  accidents. 

Our  proposed  framework  predicts  the  probability  of  ac¬ 
cidents  based  on  both  internal  and  external  information  of 
vehicles.  The  external  information  is  limited  to  the  status  of 
traffic,  i.e.,  heavy  or  low  traffic  situation.  It  does  not  consider 
the  status  of  other  drivers  in  the  nearby  vehicles.  If,  in  case, 
the  internal  situation  of  the  neighboring  vehicle’s  driver  is 
abnormal,  even  though  our  proposed  framework  will  give  a 
safe  alert,  there  may  be  a  chance  of  an  accident.  In  our  future 
work,  we  intend  to  integrate  the  nearby  vehicles’  internal 
information  so  that  our  framework  can  make  more  accurate 
predictions  of  accidents. 

V.  Conclusion 

A  probabilistic  CNN-fuzzy  logic  framework  for  accident 
prediction  in  vehicular  networks  is  proposed  in  this  study.  The 


proposed  framework  involves  human-in-the-loop  to  classify 
emotions  of  driver,  i.e.,  heartbeat  and  facial  expressions  in 
addition  to  considering  traffic  status.  The  proposed  frame¬ 
work  develops  a  probabilistic  graph-based  inference  model 
to  identify  the  probability  of  accident  by  using  the  classified 
data  of  the  initial  learning  phase.  Thereafter,  the  model  maps 
the  degree  of  severity  of  the  accident  by  using  fuzzy  rule- 
base  mapping  method.  Finally,  the  alerts  are  generated  for  the 
driver  to  take  the  necessary  precautions.  We  have  validated  the 
framework  on  the  Kaggle  data,  which  is  suitable  for  accident 
prediction  having  emotional  state  and  road  traffic  data.  The 
findings  of  the  experiments  show  that  the  proposed  framework 
outperforms  the  baseline  methods  in  effectively  predicting 
accidents  by  considering  human-in-the-loop  in  addition  to 
other  important  factors.  The  proposed  framework  shows  better 
performance  than  the  baseline  methods  in  terms  of  accuracy, 
precision,  and  FI  score.  We  intend  to  design  and  analyze 
fusion  operators  in  the  current  settings  as  part  of  the  future 
work  of  this  study. 
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